Television sets and many other video display devices, e.g., computers, support the ability to simultaneously display multiple images. In the case of television sets, the ability to display multiple images at the same time is often called picture-in-picture (PIP) capability. In the case of computer systems, the display of multiple images at the same time often occurs when multiple windows are open simultaneously in a system which supports windows based user interfaces.
In order to reduce data storage and transmission bandwidth requirements, video signals and images, e.g., digital television signals, are normally compressed prior to transmission and/or storage. Even in compressed, e.g., encoded form, video images can comprise a large amount of data. Before being displayed they must normally be decoded. Decoding of images can require a considerable amount of processing resources.
When video programs comprising sequences of images are being displayed, the images in each window need to be updated as the image sequence being displayed in a given window changes, e.g., due to motion in the images being displayed. Accordingly, to display a video sequence including motion, multiple images represented by, e.g., frames, need to be decoded and displayed over a period of time.
The number of images which are to be displayed in a given period of time for a given video image is often called the frame rate. It is frequently expressed in terms of frames per second. The time used to display a single image is called a frame time.
In the case of a television, decoder circuitry is often subject to real time processing constraints. In other words, the video images must be decoded in an amount of time equal to, or less than, the time that will be used to display the images. Failure to meet real time decoding requirements can result in the loss of data, e.g., frames, which may give the appearance of jerky motion. It may also cause other image errors and/or distortions which can result in an unpleasant and/or undesirable viewing experience.
When video sequences are being displayed in multiple windows at the same time, to support real time decoding of all the images in the video sequences being displayed, a decoder must be able to decode, on average, a number of images at a rate equal to the sum of the frame rates for each of the individual video sequences being displayed.
In addition to decoder processing resources, supporting the simultaneous display of multiple images and/or video sequences can place a considerable burden on other system resources as well. The large amount of data associated with video images which must be transferred for decoding, in some cases, can exceed a system's available bus bandwidth. This is particularly the case when multiple images are to be decoded and displayed simultaneously.
The demand for decoder processing and/or bus resources is a function of the number of images which must be decoded and displayed in a fixed amount of time. In the case of a PIP embodiment, such as a PIP capable television system, the demands on video decoder processing and bus resources can be considerable.
When more than two video sequences are displayed on a system, e.g., a television or computer, if the total load on the system resources, e.g., CPU and/or bus, required to decode and display the video sequences exceeds the available resources, images in the sequences will normally be degraded and/or discarded. The inability to fully decode images to be displayed in all of a systems open windows may result in jerky motion in one or more image sequences.
This may cause a viewer of the decoded images to feel uncomfortable. In addition, it might also make it difficult for a viewer to follow the video sequences, e.g., TV programs, being displayed. Such consequences of limited system resources could potentially defeat the purpose of displaying multiple images on a single screen.
In view of the above remarks, it is apparent that there is a need for methods and apparatus for allocating resources when multiple images, or multiple sequences of images, are to be decoded and displayed on a single display device. There is also a need for reducing or limiting the demand for decoder and display resources, e.g., to levels which are available in a given system.